Data migration

Strategies to migrate data properly

Created by Ignasi 'Iggy' Bosch / @ignasibosch

  1. Introduction
  2. MAPPING
  3. EXTRACT
  4. TRANSFORM
  5. LOAD
  6. TESTING
  7. VERIFICATION
  8. CONTINGENCIES

Structure

Automatic
name + lastname → name

Manual
name → name, lastname

Modify

Automatic
“1” → “M”, “2” → “F”,
“3” → NULL

Manual
Naming homogenize

New Data

Automatic
WHOIS, Country code etc.

Manual
Phone number, additional email etc

Use case 1:

  • Migrate system integrated into the application
  • Temporary table on the source db
  • Encrypted passwords

Use case 2:

  • 2 CSV files (customers and domains) to Postgres SQL file 
    
                        currval(pg_get_serial_sequence('user_table', 'user_id'))) returning "domain_id"
  • First test run was ~14 hours long reduced to ~4 hours
  • Interrupted & resumed

Use case 3:

  • Ensure the reliability and accuracy
  • Two different systems has to live together
  • Computational cost distribution
some Cool stuff:

Wikipedia: Data Migration
Data Migration Strategies
Martin Fowler: IncrementalMigration
Department of Education: Data Migration Roadmap
A Review on Database Migration Strategies, Techniques and Tools
IBM: Best practices for data migration
Data Conversion and Migration Strategy

Thank You

https://ignasibosch.com/talks/data-migration

me@ignasibosch.com | @ignasibosch